SOLD: Sup-Optimal Low-Rank Decomposition for Efficient Video Segmentation

Video segmentation is to partition the video into several semantically consistent spatio-temporal regions. It is a fundamental computer vision problem in many applications, such as object tracking, activity recognition, video analytics, summarization and indexing. However, there exists several remaining issues to be addressed. First, most of video segmentation methods have worse segmentation quality due to only utilizing the low-level features, which are easily contaminated by video noises and usually not powerful enough to differentiate the different semantic regions. Second, exploring the internal video statistics is indispensable to improve the segmentation performance other than employing a large number of related exemplars, which is obviously time-consuming and computationally inefficient. Third, a streaming setting for video segmentation must take into account temporal long-range relationships between voxels. Motivated by the advances in subspace clustering [4], especially the Low-Rank Representation (LRR) methods for image segmentation [1, 3], we propose a Sub-Optimal Low-rank Decomposition (SOLD) algorithm, which pursues the low-rank representation for efficient video segmentation. Instead of using superpixels in previous works like [2], we take supervoxels as graph nodes to infer their optimal affinities because they can preserve local spatio-temporal coherence as well as good boundaries. To seek the unbiased and task-independent video segmentation solution, we define our low-rank model based on very generic assumption inspired by [5]. We assume that the intra-class supervoxels are drawn from one identical low-rank feature subspace, and all supervoxels in a period lie on a union of multiple subspaces, which can be justified by natural statistic and observations of videos. Based on this assumption, the tractable low-rank representation model can be formulated as